Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework.

OBJECTIVE: Large language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT's performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare. DESIGN: We evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question difficulty and question length. Incorrectly answered questions were analysed further using a clinical reasoning framework to assess the errors made. SETTING: Online using ChatGPT web interface. PRIMARY AND SECONDARY OUTCOME MEASURES: Primary outcome was the score (percentage questions correct) in the MRCP postgraduate written examinations. Secondary outcomes were qualitative categorisation of errors using a clinical decision-making framework. RESULTS: ChatGPT achieved accuracy rates of 86.3% (part 1) and 70.3% (part 2). Weak but significant correlations were found between ChatGPT's accuracy and both just-passing rates in part 2 (r=0.34, p=0.0001) and question length in part 1 (r=-0.19, p=0.008). Eight types of error were identified, with the most frequent being factual errors, context errors and omission errors. CONCLUSION: ChatGPT performance greatly exceeded the passing mark for both exams. Multiple choice examinations provide a benchmark for LLM performance which is comparable to human demonstrations of knowledge, while also highlighting the errors LLMs make. Understanding the reasons behind ChatGPT's errors allows us to develop strategies to prevent them in medical devices that incorporate LLM technology.

Does resilience predict hospital length of stay after total knee arthroplasty? A prospective observational cohort study.

BACKGROUND: Resilience, or the ability to bounce back from stress, is a key psychological factor that is associated with ongoing functional independence and higher quality of life in older adults in the context of chronic health conditions. Emerging research has explored resilience and patient-reported outcomes after TKA. Our primary aim was to explore the relationship between resilience and acute hospital length of stay after total knee arthroplasty (TKA). METHODS: A prospective observational study recruited 75 participants one month before total knee arthroplasty from two Australian hospitals. Two preoperative psychological measures were used: the Brief Resilience Scale, and for comparison, the Depression, Anxiety and Stress Scale-21 (DASS-21). We collected sociodemographic, medical and surgical details, patient-reported pain, function, fatigue and quality of life one month before TKA. Health service data describing acute hospital length of stay, inpatient rehabilitation use, and physiotherapy occasions of service were collected after TKA. Non-parametric analysis was used to determine any differences in length of stay between those with low or high resilience and DASS-21 scores. Secondary regression analysis explored the preoperative factors affecting acute hospital length of stay. RESULTS: No significant difference was detected in length of stay between those with a low or a high resilience score before TKA. However, the group reporting psychological symptoms as measured by the DASS-21 before TKA had a significantly longer acute hospital length of stay after TKA compared to those with no psychological symptoms [median length of stay 6 (IQR 2.5) days vs. 5 (IQR 2) days, respectively (Mann-Whitney U = 495.5, P=0.03)]. Multivariate regression analysis showed that anesthetic risk score and fatigue were significant predictors of length of stay, with the overall model demonstrating significance (χ2=12.426, df = 4, P=0.014). CONCLUSIONS: No association was detected between the brief resilience score before TKA and acute hospital length of stay after TKA, however, symptoms on the DASS-21 were associated with longer acute hospital length of stay. Preoperative screening for psychological symptoms using the DASS-21 is useful for health services to identify those at higher risk of longer acute hospital length of stay after TKA.